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Abstract 

Quantization methods have been introduced to perform 
large scale approximate nearest search tasks. Resid¬ 
ual Vector Quantization (RVQ) is one of the effective 
quantization methods. RVQ uses a multi-stage code¬ 
book learning scheme to lower the quantization error 
stage by stage. However, there are two major limita¬ 
tions for RVQ when applied to on high-dimensional ap¬ 
proximate nearest neighbor search: 1. The performance 
gain diminishes quickly with added stages. 2. Encoding 
a vector with RVQ is actually NP-hard. In this paper, 
we propose an improved residual vector quantization 
(IRVQ) method, our IRVQ learns codebook with a hy¬ 
brid method of subspace clustering and warm-started k- 
means on each stage to prevent performance gain from 
dropping, and uses a multi-path encoding scheme to en¬ 
code a vector with lower distortion. Experimental re¬ 
sults on the benchmark datasets show that our method 
gives substantially improves RVQ and delivers better 
performance compared to the state-of-the-art. 


Introduction 

Nearest neighbor search is a fundamental problem in 
many computer vision applications such as image re¬ 
trieval ([RuvHuang7andT!hangT999) and image recogni¬ 
tion dLowe 19991 1. In high dimensional data-space, nearest 
neighbor search becomes very expensive due to the curse 
of dimensionality ( Indyk and Motwani 1998| l. Approximate 
nearest neighbor (ANN) search is a much more practical ap¬ 
proach. Quantization-based algorithms have recently been 
developed to perform ANN search tasks. They achieved 
superior performances against other ANN search meth¬ 
ods dJegou, Douze, and Schmid 2011 1. Product Quantiza¬ 
tion |Jegou, Douze, and Schmid 201 1) is a representative 
quantization algorithm. PQ splits the original d-dimensional 
data vector into M disjoint sub-vectors and learn M code¬ 
books {Cl ■••Cm}, vvhere each codebook contains K 
codewords Cm = {cm(l), • • ■ , Cm(Ar)}, m e 1 ■ • • M. 
Then the original data vector is approximated by the Carte¬ 
sian product of the codewords it has been assigned to. PQ 
allows fast distance computation between a quantized vec¬ 
tor X and an input query vector q via asymmetric distance 
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computation (ADC): the distances between q and all code¬ 
words Cm{k),m € 1 • • • M, k G 1 • • • AT are precomputed, 
then the approximate distance between q and x can be ef¬ 
ficiently computed by the sum of distances between q and 
codewords of x in 0(M) time. Compared to the exact dis¬ 
tance computation taking 0{d) time, the time complexity is 
drastically reduced. 

Product Quantization is based on the assumption that the 
sub-vectors are statistically mutual independent, such that 
the original vector can be effectively represented by the 
Cartesian product of quantized sub-vectors. However vec¬ 
tors in real data do not all meet that assumption. Optimized 
Product Quantization (OPQ) (IGe et al. 20131 1 and Cartesian 
K-means (INorouzi and Fleet 20131 1 are proposed to find an 
optimal subspace decomposition to overcome this issue. 

Residual Vector Quantization (RVQ) 
( |Chen, Guan, and Wang 2010) is an alternative ap¬ 
proach to perform approximate nearest neighbor 
search task. Similar to Additive Quantization (AQ) 
(jBabenko and Lempitsky 2014) and Composite Quantiza- 
tion ( [Ting Zhang 20 f4| l, RVQ approximates the original 
vector as the sum of codewords instead of Cartesian prod¬ 
uct. Asymmetric distance computation can also be applied 
to data quantized by RVQ. RVQ adopts a multi-stage 
clustering scheme, on each stage the residual vectors are 
clustered instead of a segment of the original vector. Com¬ 
pared to PQ, RVQ naturally produces mutually independent 
codebooks. However, the gain of adding an additional stage 
drops quickly as residual vectors become more random, 
limiting the effectiveness of multi-stage methods to only a 
few stages ( [Gersho and Gray 1992| l. A direct observation is 
that the encodings of codebooks learned on the latter stages 
have low information entropy. Moreover, encoding a vector 
with dictionaries learned by RVQ is essentially a high-order 
Markov random field problem, which is NP-hard. 

In this paper, we propose the Improved Residual Vector 
Quantization (IRVQ). IRVQ uses a hybrid method of sub¬ 
spaces clustering and warm-started k-means to obtain high 
information entropy for each codebook, and uses a multi- 
path search method to obtain a better encoding. The basic 
idea behind IRVQ is rather simple: 

1. Subspace clustering generally produces high information 

entropy codebook. Though we seek a clustering on the 

whole feature space, such codebook is still useful. We 
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Utilize these information by warm-start k-means with this 
codebook. 


2. The norms of codewords reduce stage by stage. Though 
the naive ’’greedy” encoding fails to produce optimal en¬ 
coding, a less ’’greedy” encoding is more likely to obtain 
the optimal encoding. We propose a multi-path encoding 
algorithm for learn codebooks. 


The codebooks learned by IRVQ are mutually indepen¬ 
dent and each codebook has high information entropy. And 
a signihcantly lower quantization error observed compared 
to RVQ and other state-of-the-art methods. We have val¬ 
idated our method on two commonly used datasets for 
evaluating ANN search performance; SIFT-IM and GIST¬ 
IM (Jegou, Douze, and Schmid 2011 1 . The empirical results 
show that our IRVQ improves RVQ significantly. Our IRVQ 
also outperforms other state-of-the-art quantization methods 
such as PQ, OPQ, and AQ. 


Residual Vector Quantization 

Residual vector quantization (RVQ) 

( Juang and Gray Jr 1982| l is a common technique to 
approximate original data with several low complexity 
quantizers, instead of a prohibitive high complexity quan¬ 
tizer. RVQ reduces the quantization error by learning 
quantizers on the residues. RVQ is introduced to perform 
ANN-search in ( |Chen, Guan, and Wang 2010 1, The gain 
of adding an additional stage relies on the commonality 
among residual vectors from different cluster centers. Thus 
on high-dimensional data this approach performs badly. 


Information Entropy 

It has been observed that the residual vectors become very 
random with increasing stages, limiting the effectiveness of 
RVQ to a small number of stages. To begin with, we first 
examine the encoded dataset by RVQ from the point of view 
of information entropy. 

For hashing based approximate nearest neigh¬ 
bor search methods, e.g. Spectral Hashing 
( |Weiss, Torralba, and Fergus 2009| ), we seek a code 
that each bit has a 50% chance of being one or zero, 
and different bits are mutually independent. Similarly, 
we would like to obtain maximum information entropy 
S{Cm), defined below, for each codebook and no mutual 
information between different codebooks. 


K 

s{c^) = '£pT{\og^pT) = iog^K 

k^l 

n 7 Pijiki,kj) ( 1 ) 

Py [ki , kj) log 2 —— = 0 

ki.kjGl-K PkiPkj 

for i,j G 1 • • ■ M 

where p™ denotes the probability that in the dictio¬ 

nary Cm, its fc-th element is chosen; and pij{ki,kj) de¬ 
notes the probability that ki-th element from and kj- 
th element from Cj are chosen by a vector x simultane¬ 
ously. In Fig. |2] we drew the information entropy at each 



Figure 2: Entropy of each stage on SIFTIM and GISTIM 
datasets. Learned with k — 256. For the codebook learned 
on stage m, information entropy E — Y^=iPn(\og 2 Pn), 
where p„ = P{im{.^) = n). 


stage of the RVQ to reveal a phenomenon: For SIFT-IM 
dataset ( |Jegou, Douze, and Schmid 201 1| , the entropy drops 
to 5.Shits at the 16th stage, and 4.34bits for GIST-IM. That 
means though we wish to learn K codewords for each code¬ 
book, only a few of them are effective. Nevertheless, quan¬ 
tization on residual vectors leads to a much lower mutual 
information compared to PQ because residues are largely 
independent as shown in FiglT] 

Encoding 

RVQ encodes according to the distance between current 
residual vector and each codewords. We shall call it Se¬ 
quential Encoding, which is actually a greedy algorithm. Eor 
codebooks learned with RVQ, The codewords chosen on a 
stage will affect the choice on the latter stages. Consider en¬ 
coding a vector x with M codebooks learned, the quantiza¬ 
tion etTor is: 


M' 

■^ = X! 11^ “ c^(*m(x))|p - (m - l)||xf 

, , ( 2 ) 
+ X 1 Ca(ia(x))^C6(4(x)) 

b—l,a^b 

Denote e as the third term. Sequential Encoding cannot 
predetermine the value of e. Eor PQ-based schemes such 
as Product Quantization ( | Jegou, Dou ze, and Sc hmid 201 1| ), 
Optimized Product Quantization dGe et al. 2013b . such min- 
imization scheme guarantees finding the best encoding, 
since codewords from different codebooks are mutually or¬ 
thogonal. However, for RVQ, Sequential Encoding is not op¬ 
timal. The encoding with codebooks learned by RVQ is ac¬ 
tually a fully connected discrete pairwise MRE energy opti¬ 
mization, which is actually NP-hard. 

In addition, the quantization errors are propagated to the 
next stage in the codebook learning process. RVQ actually 
uses the information of the encoded vectors on its learning 
phase; the residual vectors are determined by both code¬ 
books and the encoding mechanism. As RVQ for ANN 
search requires many stages of codebook learning, the ac- 


























(a) Product Quantization 


(b) Residual Vector Quantiza- (c) Improved Residual Vector (d) Addtive Quantization 
tion Quantization(/ = 10, L = 10) 


Figure 1; Mutual information matrix for different quantization methods. Experiment conducted on a GIST-IM dataset. We 
learned M — 8 codebooks, K — 256 codewords per codebook. The perfect encoding should have no mutual information 
between different codebooks and an information entropy of log K = 8-bits for each codebook. Our proposed IRVQ achieves 
near optimal encoding. 


cumulated quantization loss could become very high with 
Sequential Encoding. 


Improved Residual Vector Quantization 

As we have presented the two major drawbacks of RVQ. 
In this section, we propose our Improved Revised Residual 
Vector Quantization (IRVQ) to overcome these issues. IRVQ 
improves RVQ in three aspects: 

Improved Codebook Learning Instead of performing 
cold-started k-means as in RVQ, we warm-start k-means 
with initial codebook learned on a relatively smaller 
subspace. 

Multi-path Vector Encoding In addition to assigning a 
vector to a single codeword, multiple codewords may be 
kept as candidates. 

Joint Optimization On each stage, we first use Improved 
Codebook Learning to learn a codebook, then use Multi- 
path Vector Encoding to encode the dataset vectors, so the 
residual vectors could be even smaller. 

See Algorithm [T] for the full pseudo code of IRVQ. The 
information entropy is maximized with our Improved Code¬ 
book Learning as shown in Eigure |2] and with Multi-path 
Vector Encoding better approximations of original vectors 
are found as shown in Eigure [3] We jointly use Improved 
Codebook Learning and Multi-path Vector Encoding to fully 
optimize the a stage, and the quantization error is even low¬ 
ered. We shall explain our algorithm in more details in the 
following text. 


Improved Codebook Learning (ICL) 

To obtain better clustering performance on high dimen¬ 
sional data, one of the popular approaches is to cluster 
on lower-dimensional subspace ( Agrawal et al. 1998) l, 
this is also what PQ/OPQ do to obtain high infor- 
mation entropy for each dictionary. Many previously 
proposed methods for high dimensional data clus- 
tering, e.g. ( Bouve yron, Girard, and Schmid 2007| l, 
( ling, Ng, and Huang 2007) ^ seek clustering in an op¬ 
timal subspace instead of the whole feature space. In 
lower-dimensional subspaces the projected datasets become 


Algorithm 1 Improved Residual Vector Quantization 
Input: Training samples X of d-dimension, number of 
stages M, number of centroids to learn on each stage K, 
best L approximations on each stage, I iterations. 

Output: Codebooks: = {cm(l • ■ ■ K), m = 1 • • • M}. 

1: Initialize Residue: R = X 

2: Denote the best L approximations for x on stage m: 
1---L} 

3: for m = 1 • ■ • M do 

4: Perform PCA on R and extract principal components 

(ri,r2,--- ,rd). 

5: ^ k-means on (ri, r 2 , • • ■ , j^R 

6: for p = 2 • • • / do 

7: Y ^ (ri,r2,--- ,rprfp//i)'^R 

8: k-means on Y: 

Initialize k-means algorithm with C^, missing 
dimensions are padded with zeros. 

9: end for 

10: Cjn — AC^ is the codebook for stage m. 

11: for all X s X do 

12: find the best L approximations of x ss xj„ = 

x^_i -f Cm(fc), V = I - ■■ L,k = 1- ■■ K. 

13: end for 

14: r = X - x^ 

15: end for 


denser and then a balanced clustering could be easily 
obtained. However, in the case of fitting the residual vectors, 
it’s not reasonable to clustering on just a few dimensions as 
the residue vectors lies in the whole feature space randomly. 
We thus seek a hybrid way to perform clustering on the 
residual vectors. 

We propose Improved Codebook Learning (ICL) to this 
end. It is done by iteratively adding dimensions for clus¬ 
tering. Eirst designate a dimensions adding sequenc: di < 
d 2 < ■ ■ ■ < dj = d, then, then: 

Initialization We first perform the principle component 
analysis (PCA) on the residual vectors R and extract the 
principal dimensions. 



























Squared Distortion on each stage, SIFTIM 


Iterative Warm-start On the i-th iteration, perform k- 
means on top di dimensions, initialized with previous 
learn codeword^ Note that the codewords have missing 
dimensions, we simply pad zeros to them. 

Restoration Transform the codebook in PCA dimensions 
back to the original data-space dimensions. Then the re¬ 
sulting codebook can be used in the same way as the code¬ 
books learned with usual k-means algorithm. 


There are several motivations for our improved codebook 
learning method: 


1. As depicted in (Ding and He 2004 1 , PCA dimension re¬ 
duction finds the best low-dimensional L2 approximation 
of the data. 


2. Finding a better initial points for k-means almost always 
leads to a quicker convergence and a better clustering 
( Bradley and Fayyad 1998[ ). As padding extra dimensions 
with zeros doesn’t affect the encoding, it’s reasonable to 
infer that the iterative initialization could also lead to bet¬ 
ter clustering. 

3. PCA dimensions are essentially rotation of original data 
with no loss of information or metric change. Padding ze¬ 
ros doesn’t change the clustering results. 

We warm-start k-means on the most significant PCA dimen¬ 
sions so clustering on the higher dimensions could started 
from better initial positions. In our experiment, a very low 
/ (in our experiments, /=10 at most), could already achieve 
satisfying results. By performing ICL, As shown in Figure 
|2] the entropy of each codebook is maximized. 


Multi-path Vector Encoding (MVE) 

Encoding for Product Quantization is quite simple 
since the original feature space has been divided 
into mutually orthogonal subspaces. However, for 
Additive Quantization dBarnes and Frost 1993k Com¬ 
posite Quantization ( Ting Zhang 2014| l, and Residual 
Vector Quantization, the encoding is an MRF prob¬ 
lem as mentioned in Section „ Additive Quantization 
( [Babenko and Lempitsky 2014) l proposed a Beam Search 
algorithm in a matching pursuit fashion, however, runs 
slowly in practice( {Babenko and Lempitsky 2015] l. 

Suppose the best approximation (correct encoding) of an 
input vector is x ss ci(ji) -|- 02 ( 12 ) + • • • + CMiiAi)- Fur¬ 
ther assume we have known the first m — 1 correct encod¬ 
ings ii, 12 , • ■ • , ifn-i, can we effectively compute De¬ 
note the known part as x = Ci(ii) + ■ ■ • + Cm-i(*m-i) and 

the unknown part as x= Cm+i{im+i) + • • • + CM(iM), we 
seek the correct encoding on the m-th dictionary im- Notice 
that: 


||X-X - Cmiim) - x'lP = ||x - x|p -f ||x- x|p -f 2x'^ X 
4~ ||x C 7 ^(?j 77 ,)|| 2x 2c^(z^)) X 

-2||xf 

(3) 


*On the first iteration we directly learn a codebook 



Stages 


Squared Distortion on each stage, GISTIM 



Figure 3: Distortion on SIFTIM and GISTIM, with M = 
16, k = 256, also compared with Optimized Product Quan¬ 
tization. Note that L=10 7=1: use MVE only; L=l, 1=10: 
use ICL only; L=10, 7=10: Jointly use MVE and ICL 


The first three terms can be seen as constants when we 
seek the correct , and the last term can be omitted. The 
fourth and fifth term can be effectively computed. However 
the sixth term cannot be computed because we don’t know 

X. If we omit this term extra error will be introduced. To 
lessen this error, we hope ||x|| is very small so that the vari¬ 
ance of the last term won’t have an serious impact on the 
final outcome. The norms of codewords from codebooks 
learned with IRVQ naturally shrinks, so ||x|| is decreased 
stage by stage. 

Thus we maintain a list of best L approxi¬ 
mations of X with the first (m — 1) codebooks: 

■ ■ ■ ,x(„_]^}. Then we encode with the 
next codebook Cm = {cm (1) (2) (77)}. We 

find L combinations from {x.m_i+Cmik)}, I € 1 ■ ■ ■ L,k G 
1 ■ ■ ■ K minimizing the following objective function: 

||x - x(„_i - C„(fc)f =||x - x(„f -f ||x - Cmik)f 

-M^ + 2Cm{kf^T~" 

(4) 

The first term has been computed at the previous en¬ 
coding step, and the third term ||x|p is constant for any 

{:>4n-i + Cm{k)), is thus negligible. And the last term in¬ 
volves m table lookups and addition, with the inner-product 
of all codewords precomputed before MVE. Thus, only the 
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Figure 4; How quantization on e affects ANN search perfor- 
mance(Measured by Recall® 1). Experimented on SIFT-IM 
and GIST-IM datasets, with M = 8,k = 256, L = 30 ,1 = 
10 


term ||x — Cm(fc)|P is required to be computed. The time 
complexity is 0{dK + mKL + KL log L) for encoding with 
one single dictionary on the m-th stage. 

To sum up, MVE iteratively uses the top L candidates as 
seeds to find the best encoding for x. The resulting method 
is quite similar to the multi-path search for residual tree 
( jKossentini, Smith, and Barnes 1992) , which is used on sig¬ 
nal reconstruction. Note that MVE degrades to Sequential 
Encoding when L = 1. 

Learning codebooks jointly with Improved 
Codebook Learning and Multi-path Encoding 

Remind that RVQ is a multiple stage learning procedure, 
at each stage the quantization error is left to the next it¬ 
eration. This makes reducing quantization error for every 
single stage necessary. Now we have already presented the 
Improved Codebook Learning for learning balanced code¬ 
words, and Multi-path Vector Encoding for better encoding 
the original vector, we next jointly use them to optimize a 
stage in IRVQ. At each stage, our IRVQ does the following: 

1. Use the Improved Codebook Learning to learn a code¬ 
book; 

2. Use the Multi-path Vector Encoding to find a best ap¬ 
proximation of the original vector with all the codebooks 
learned. 

Note that for each stage, the L best approximations gener¬ 
ated by MVE can be stored for future use, so the MVE on 
next stage only requires to encode one more codebook. 

Further Lower the Memory Consumption 

One of the advantage of quantization methods for ANN 
search is that vectors are compressed into a few bits so an 
in-memory search is feasible. Take PQ as an example, for 
an encoding with M = 8, K = 256, an original vector 
could be compressed into M log 2 K — 64-bit code, thus a 
dataset containing 1 billion vectors can be fully loaded into 
a 8G RAM. Eor RVQ, AQ and our IRVQ, an extra storage 
overhead is required. Consider the expansion of distance be- 




(a) Recall@4 on SIFTIM, (b) Recall@4 on GISTIM, 
different code length different code length 
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Eigure 6: Effect of different code length for different ANN 
search methods. Experimented on datasets SIETIM and 
GISTIM. The recall of the true neighbor in the top 4 ranked 
quantized element (Recall @4) is used to measure the ANN 
search quality. 


tween the approximated vector x and the query vector q: 

M 

l|q - xf = X! Ilq “ c^(tm(x)))f - (to - l)||qf -F e 

m—1 
M M 

e = XI X Ca(fa(x))'^Cf,(4(x)) 

a—1 b—l,b^a 

(5) 

Note that e is irrelevant to query q, so we compute and 
store e along with each quantized vector when the whole 
database is quantized, and this e causes extra memory con¬ 
sumption. An IEEE-754 floating point number takes 32-bit 
of memory. However, it’s possible to quantize e into a few 
bits. Eigure m shows that using only 8-bits of extra storage 
could already preserve the searching quality. 

Eor real world ANN search applications, the major con¬ 
cern is the response speed instead of the memory consump¬ 
tion. Since an additional table look-up (indirect addressing) 
takes much more time compared to memory copy, if mem¬ 
ory size is not an issue we recommend not to quantize e for 
fastest speed. 

Approximate Nearest Neighbor Search 
Performance 

We performed the ANN search tests on the two 
datasets commonly used to validate the efficiency 
of ANN methods: SIET-IM and GIST-IM from 
( |Jegou, Douze, and Schmid 201 1) : 

SIFTIM contains one million of 128-d SILT (ILowe 20041) 
features. It’s commonly used with local feature descriptor 
for various image related applications. 

GI STIM contains one million of 960-d GIST 
(lOIiva and Torralba 200 II I global descriptors. 

Eor each dataset, we randomly pick 100,000 vectors as the 
training set. We then encode the rest of the database vectors, 
and perform 1000 queries to check ANN search quality. 
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Figure 5: The performance for different algorithms on SIFT-IM and GIST-IM, with 128-bit encoding(M=16) 


Evaluated Methods 


We compared our IRVQ to the following state-of-the-art 
quantization methods: 

PQ : Product quantization proposed in 

I. Following 

I, we used the struc¬ 
tured ordering for GIST-IM and the natural ordering for 
SIFT-IM. 


(Jegou, Douze, and Schmid 201 
(Jegou, Douze, and Schmid 201 


OPQ : Optimized Product Quantization proposed in 
(IGe et al. 20131) . We adopted the non-parametric version 
of OPQ. Cartesian k-means, the algorithm proposed in 
dNorouzi and Fleet 2013l l shares a similar idea and has the 
same performance with OPQ. 


AQj _ Additive Quantization 

(Babenko andLempitsky 2014 1 . We adopted the pa¬ 
rameters suggest in the paper. 


R VQ : Residual Vector Quantization proposed in 

(Chen, Guan, and Wang 20101. 

For our IRVQ we set the parameters as / = 10, L = 
30. For all the methods, we choose K = 256 as the 
size of each codebook. We perform linear scan search 
with asymmetric distances computation (ADC) proposed in 
(Jegou, Douze, and Schmid 2011| , which directly compares 
the input query and the quantized dataset. The search qual¬ 
ity is measured using recall @i?, which means that for each 
query, we retrieved R nearest items and check whether they 


contain the true nearest neighbor. Such criterion is com¬ 
monly used to evaluate the ANN methods. 

Results 

Figure |5] shows the recall curve of various state-of-the-art 
methods. Our IRVQ improves RVQ significantly, for exam¬ 
ple, on 64bit encoding, IRVQ obtained 58.31% recall@4 
for SIFTIM, while plain RVQ is only 50.35%, the rela¬ 
tive improvement is 15.8%. The improvement is even sig¬ 
nificant on higher dimensional data GISTIM, where IRVQ 
gained 28.4% and RVQ gained 18.6% accuracy, relatively 
52.7% improvement on recall@4. IRVQ also outperforms 
other state-of-the art, for example, IRVQ outperforms AQ 
by 17.7% on the recall® 1 for 64bit SIFTIM encoding. 

We can also see by the results, RVQ doesn’t perform well 
with added stages. On lower bits and low dimensions RVQ 
has advantages over PQ/OPQ, however, on higher bits or 
higher dimensions plain RVQ performs badly. Fig|6] illus¬ 
trates the effect of code length for different methods. Our 
IRVQ fixed the problem and deliver consistently high per¬ 
formance gain with added stages. 

Conclusion 

In this paper, we proposed the Improved Residue Vector 
Quantization (IRVQ) for large-scale high-dimensional ap¬ 
proximate nearest neighbor search. We proposed Improved 
Codebook Learning (RCL) and the Multi-path Vector En¬ 
coding (MVE) to deliver consistent performance gain with 

























adding stages. Experiment against several state-of-the-art 
quantization methods on two well known dataset demon¬ 
strate the effectiveness of IRVQ. 
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